Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core][state] Proper report of failure when job finishes #31761

Merged
merged 24 commits into from
Jan 23, 2023
Merged

[core][state] Proper report of failure when job finishes #31761

merged 24 commits into from
Jan 23, 2023

Conversation

rickyyx
Copy link
Contributor

@rickyyx rickyyx commented Jan 18, 2023 •
edited
Loading

Why are these changes needed?

This PR handles cases when a job finishes but tasks still running should be marked as failed.

  • It adds a handler function OnJobFinished as a job finish listener in the GcsJobManager, so when a job is marked as finished, the OnJobFinished will be called to mark any non-terminated tasks as failed

Related issue number

Checks

@rickyyx
init
547177a 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
comment
60c080b 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
typo
5770dcc 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
change index
c93eaa8 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
used absl::btree_multimap
42c3bf2 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
address comments
732d728 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
update
e4dc1f1 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
nit
2a3b92b 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
update
72f5212 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
comments on parent task id and detached actor
a9e8b55 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
Merge branch 'master' of github.com:ray-project/ray into task-backend…
303bd4a 
…-gcs-mark-failures
@rickyyx
Merge branch 'master' of https://github.com/ray-project/ray into task…
fe92e82 
…-backend-job-fail
@rickyyx
tests ok
170ac37 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
format
14caabd 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
typo
7f1fa5d 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
line
a9b1526 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
comments
bac9fe7 
Signed-off-by: rickyyx <[email protected]>
@rickyyx rickyyx marked this pull request as ready for review January 19, 2023 07:26
rkooo567
rkooo567 reviewed Jan 19, 2023
View reviewed changes
python/ray/tests/test_task_events.py Outdated Show resolved Hide resolved
src/ray/common/ray_config_def.h Outdated Show resolved Hide resolved
src/ray/gcs/gcs_server/gcs_server.cc Outdated Show resolved Hide resolved
src/ray/gcs/gcs_server/test/gcs_task_manager_test.cc Outdated Show resolved Hide resolved
src/ray/gcs/gcs_server/gcs_task_manager.cc Show resolved Hide resolved
src/ray/gcs/gcs_server/test/gcs_task_manager_test.cc Outdated Show resolved Hide resolved
src/ray/gcs/gcs_server/gcs_task_manager.cc Outdated Show resolved Hide resolved
@rkooo567 rkooo567 added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jan 19, 2023
@rickyyx
clear
fa72efc 
Signed-off-by: rickyyx <[email protected]>
@rickyyx rickyyx removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jan 20, 2023
@rickyyx
const
0443615 
Signed-off-by: rickyyx <[email protected]>
@rkooo567 rkooo567 added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jan 20, 2023
@rickyyx rickyyx removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jan 20, 2023
rkooo567
rkooo567 approved these changes Jan 23, 2023
View reviewed changes
Copy link
Contributor

@rkooo567 rkooo567 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Two last requests!

python/ray/tests/test_task_events.py Show resolved Hide resolved
src/ray/gcs/gcs_server/gcs_task_manager.cc Show resolved Hide resolved
@rkooo567 rkooo567 merged commit 86bd6c6 into ray-project:master Jan 23, 2023
@rkooo567
Copy link
Contributor

Btw note; the PR description may be wrong. We only mark children failed (not grandchildren)

@rkooo567
Copy link
Contributor

Let's also close the P0 issue!

@rickyyx rickyyx changed the title [core][state] Proper report of failure when job finishes and for finished tasks [core][state] Proper report of failure when job finishes Jan 23, 2023
@rickyyx
Copy link
Contributor Author

rickyyx commented Jan 23, 2023

Updated PR description and title.

@rkooo567
Copy link
Contributor

Let's also make sure to not forget the follow up test!

@rickyyx
Copy link
Contributor Author

rickyyx commented Jan 24, 2023

Yep here: #31875

@rickyyx rickyyx deleted the task-backend-job-fail branch January 24, 2023 04:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rkooo567 rkooo567 rkooo567 approved these changes

@wuisawesome wuisawesome Awaiting requested review from wuisawesome

@ericl ericl Awaiting requested review from ericl

@AmeerHajAli AmeerHajAli Awaiting requested review from AmeerHajAli

@robertnishihara robertnishihara Awaiting requested review from robertnishihara

@pcmoritz pcmoritz Awaiting requested review from pcmoritz

@raulchen raulchen Awaiting requested review from raulchen

@fishbone fishbone Awaiting requested review from fishbone

@scv119 scv119 Awaiting requested review from scv119

Assignees

@rkooo567 rkooo567

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rickyyx @rkooo567